Skip to content

feat: import missing backup records from S3-compatible object storage#2333

Open
qqkzlm wants to merge 2 commits into
Wei-Shaw:mainfrom
qqkzlm:feature/import-s3-backup-records
Open

feat: import missing backup records from S3-compatible object storage#2333
qqkzlm wants to merge 2 commits into
Wei-Shaw:mainfrom
qqkzlm:feature/import-s3-backup-records

Conversation

@qqkzlm
Copy link
Copy Markdown

@qqkzlm qqkzlm commented May 10, 2026

feat: import missing backup records from S3-compatible object storage

Summary

Add an admin-triggered workflow to rebuild missing official backup records from S3-compatible object storage.

This fixes cases where backup files already exist in the configured backup bucket/prefix, but corresponding backup_records rows are missing after migration, manual upload, old deployments, or database recovery.

Changes

  • Add backup import preview API: GET /api/v1/admin/backups/import-preview.
  • Add import API: POST /api/v1/admin/backups/import.
  • Add admin UI to scan object storage first, then confirm import.
  • Import only missing .sql.gz backup objects and skip records that already exist.
  • Create imported records with status=completed, backup_type=postgres, and triggered_by=imported.
  • Store file_name, s3_key, size_bytes, and fallback timestamps from object last modified time.
  • Set expires_at for imported records based on the current backup retention policy.
  • Keep imported historical records out of automatic count-based cleanup.
  • Keep existing backup download, delete, and restore flows unchanged.
  • Add pg_dump and psql to runtime Docker images and make restore fail fast with ON_ERROR_STOP=1.

Tests

  • go test -tags unit ./internal/service -run 'TestBackupService_ImportMissingBackups|TestBackupService_ImportMissingBackups_BackfillsImportedExpiry|TestCleanupOldBackups_KeepsImportedRecords|TestStartRestore_Async|TestStartRestore_AsyncFailure' -v
  • docker build --target frontend-builder -f Dockerfile .
  • docker build -f Dockerfile .
  • docker build -f deploy/Dockerfile .
  • docker build -f backend/Dockerfile backend

Manual Verification

  • Verified locally with Postgres, Redis, MinIO, and the app container.
  • Verified on NAS with existing object-storage backups:
  • Preview found object-storage backup candidates.
  • Import created missing official backup_records.
  • Imported records appeared in the official backup list.
  • New backup restore completed successfully after adding psql to the runtime image.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 10, 2026

All contributors have signed the CLA. ✅
Posted by the CLA Assistant Lite bot.

@qqkzlm
Copy link
Copy Markdown
Author

qqkzlm commented May 10, 2026

I have read the CLA Document and I hereby sign the CLA

@qqkzlm
Copy link
Copy Markdown
Author

qqkzlm commented May 10, 2026

recheck

github-actions Bot added a commit that referenced this pull request May 10, 2026
@qqkzlm
Copy link
Copy Markdown
Author

qqkzlm commented May 12, 2026

追加了一个小修复,处理数据库恢复后旧 backup_records 被一起恢复导致后台持续轮询的问题。

场景是:恢复数据库后,备份文件里的历史 backup_records 可能包含旧的 status=runningrestore_status=running 记录。前端重新加载列表后会继续识别这些记录为进行中状态,从而一直轮询。

现在恢复成功后会复用启动时的 stale record 清理逻辑,把这些恢复出来的历史 running 记录标记为 failed,错误原因记录为 interrupted by database restore。这样不会影响已完成的备份记录,也能避免后台页面无限显示进行中。

新增测试覆盖了这个场景:恢复过程中旧 backup_records 被写回数据库,恢复完成后旧 running 记录会自动变为 failed,当前恢复记录仍保持 completed。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant